In recent years, studies about mixed reality that aims at seamless joint of real and virtual spaces have been extensively made. For example, as described in Japanese Patent Laid-Open No. 11-88913 (U.S. Pat. No. 6,522,312), a mixed reality system that presents mixed reality is implemented as a system which presents, to an observer, a mixed reality space image formed by superimposing an image of a virtual space (a virtual object generated by rendering a three-dimensional (3D) model) by computer graphics (CG), text information, and the like onto an image of a real space captured by an image sensing device such as a video camera or the like.
Conventionally, as such mixed reality system, a system that superimposes a CG model as an image of a virtual space onto a miniature as a real object located (present) on a real space is known.
This conventional mixed reality system superimposes an annotation that explains components of the CG model and miniature onto the miniature.
Furthermore, the present application has proposed by Japanese Application No. 2002-95535 (U.S. Patent Pub. No. US2003/185416 A1) a technique for realizing natural display in consideration of the depth (occlusion) so as to prevent an observer's hand from being occluded by a CG model if the observer's hand is located in front of the CG model (i.e., so as not to display the CG model on the region of the hand of the observer).
However, in the conventional mixed reality system when display is made to prevent the observer's hand from being occluded by the CD model, not only the CG model but also an annotation undergo similar display control. For this reason, when the observer's hand overlaps an annotation display, the annotation display of the overlapping portion cannot be observed.
Such annotation often includes information required for the observer to experience mixed reality. If the annotation is occluded, it is inconvenient for the observer. Especially, when comments about the types and operation methods of buttons to be operated by the observer are displayed as an annotation, this problem becomes conspicuous.
Therefore, it is demanded to attain both natural display and convenient display (i.e., the CG model is occluded by the hand but the annotation is not occluded by the hand). However, no such conventional technique is available.